Outcomes-Based Courses: Using Pre and Post Course Measurements to Examine and Enhance Course Success

Questionnaire-based pre/post course testing can be used to extend the cognitive processes provided in short (2to 6-hour) courses. When participants participate in preand post-course assessments and implement the changes they commit to at the end of a course, course designers can have an impact on knowledge and workplace practices. A carefully designed pre-course assessment offered prior to an educational activity can help professionals draw upon their workplace experiences prior to a course and during that course. Jocelyn Lockyer, PhD, Director, Continuing Medical Education and Professional Development, Associate Professor, Department of Community Health Sciences, Faculty of Medicine, University of Calgary. Richard Ward, MD FCFP, is Senior Consultant, MAINPROCourses, Continuing Medical Education and Professional Development, and Clinical Associate Professor, Department of Family Medicine, Faculty of Medicine, University of Calgary Herta Fidler, MSc, Research Associate, Continuing Medical Education and Professional Development, Faculty of Medicine, University of Calgary John Toews, MD FRCPC, Associate Dean, Continuing Medical Education and Professional Development and Professor, Department of Psychiatry, Faculty of Medicine, University of Calgary Margaret Churcher, MD CCFP, Consultant, MAINPRO-C Courses, Continuing Medical Education and Professional Development, Faculty of Medicine, University of Calgary Background It is assumed that professionals learn and gain information from continuing professional development programs (Daley, 2002). However, studies involving randomized control trials of continuing medical education (CME) programs show the results of short courses can be variable (Davis, Thomson, Oxman, & Haynes, 1995; Davis, Thomson, Freemantle, Wolf, Mazmanian, & Taylor-Vaisey, 1999). These systematic reviews of the literature suggest that short courses, particularly those that are didactic (lecture based), often fail to show change in physician behavior or patient outcomes. The ones that do show improvement use multiple learning methods; involve enabling and reinforcing strategies as well as facilitating strategies; and have a needs assessment that was used to determine the gap between current and desired clinical practice. Outside of medicine, there are even fewer studies and little attempt to measure changes in outcomes from short course curriculum (Greiner and Knebel, 2003). Daley (2002) has theorized that, while continuing professional development (CPD) programs play a vital role in providing professionals with up to date information, the context in which the professional practices frames what they learn and how they use the information in practice. Proponents of a situated view of learning argue that one cannot separate the learning process from the situation in which the learning is presented. Knowledge and the process of Journal of Executive Education Fall 2003 31 learning within this framework are viewed as a product of the activity, context, and culture in which it is developed and used (Brown, Collins, & Duguid, 1989). One of the most powerful methods by which this is attained is through cognitive apprenticeships. For example, Brandt, Farmer and Buckmaster (1993) suggest that students learn through five sequential phases: Modeling, in which they observe more experienced members of the community or hear about the ‘tricks of the trade’; Approximating, in which they try out an activity while articulating and reflecting on what they have seen and done; Scaffolding, in which coaching and teaching aids assist in developing competency, while support gradually drops away; Self-directed learning, in which students assume responsibility for the learning; and Generalizing, in which students discuss and relate what they have learned to subsequent practice situations. In medicine, this is done through problembased or case-based learning in which a patient’s case is presented and physicians discuss how they would manage that case. In this model, physicians might be explicitly asked to state the words they would use to query why the patient had come for the visit or the patient’s understanding of medications prescribed. Roleplaying may be used to ensure the student understands how to use a specific tool. For example, we explicitly taught physicians how to do the mini mental state examination (MMSE), by a role-playing exercise. Such enabling experiences try to provide authentic and not decontextualized contexts for the learning. While we accept that situated cognition strategies are useful when professionals are attending an educational event, very often the events are short (i.e., two to four hours). This time may be insufficient to ensure the types of reflection that are so necessary to learn new skills and approaches. It may not allow the learner to generalize to other patient cases that are similar but have unique nuances. Practice and feedback are often limited. We posit that a carefully designed pre-course assessment offered prior to an educational activity can help professionals draw upon their workplace experiences prior to a course and during that course. A pre-test can situate the learning early by focusing on the salient aspects of the educational program. That same test, readministered one to three months after an event, can stimulate reflection on the activity and the learner’s ability to apply this activity in the workplace. It may help the learner uncover barriers to the application of newly acquired approaches. Further, the data that are generated from preand post-testing can be useful in assessing a course’s worth, altering the design, and creating evidence that the educational unit is performing appropriately. This paper will offer some general guidelines on the development of preand post-testing, describe the experiences at the University of Calgary in the area of preand post-course testing for courses offered to family physicians, and finally offer some suggestions related to research that might be done to further establish the value of preand post-testing as an integral part of a continuing professional development strategy in other professions. PreAnd Post-Testing There are no hard and fast rules related to the approaches one might take with preand post-testing. Some favor a strictly open-ended qualitative approach to the pre-course assessment in which learners will describe their 32 Fall 2003 Journal of Executive Education intended learning expectations, barriers to learning, the role(s) they play in this area, and even the complexities of their practice vis-à-vis the learning tasks. Others who work in environments in which quantitative data are preferred and understood by decision makers may prefer to take a more quantitative approach to the pre-course. Such an approach may use multiple-choice testing to assess the learner’s knowledge base or scales (Likert-type or agreement-disagreement) to examine learner comfort or approaches to practice. The approach to questioning should be tied to the goals of the educational activity to enable students to prepare for the program and teachers to use the data to guide their pre-course preparation. Similarly at the post-course phase, one to three months later, a variety of approaches may be taken depending on the desired strategy and use of the data. For example, it may be appropriate to continue with an open-ended approach to facilitate open reflection related to content and skills learned and used since the course, barriers encountered in applying the new knowledge in the workplace, and intended further learning. This line of questioning will help learners think about their progress since the course and set goals for the future. It may be equally appropriate to repeat a quantitative precourse assessment to examine change since the course. In some cases, a combination of quantitative and qualitative data may be helpful to learner, teacher and the educational unit. The timing of the preand post-course assessments needs to be considered carefully. Too much time in advance of a course may mean that the professional’s goals and expectations have shifted. Even environmental barriers may be reduced or heightened prior to a course. However, too short a time reduces the teacher’s ability to receive and use the data in a meaningful way. Post-course assessment has its challenges as well. Too little time may mean the professional has not had a chance to practice the new approach. Too much time may limit the learner’s ability to recall the salient aspects of the course and its application to practice. Data that are generated in such circumstances may not be valid. Certainly, the validity of self-report data has been questioned particularly when objective examinations of medical records fail to provide the evidence that the desired change has taken place despite self-report data to the contrary (Adams, Soumerai, Lomas, & Ross-Degnan, 1999). The University Of Calgary Approach And Findings The University of Calgary Office of Continuing Medical Education and Professional Development regularly uses a pre/post assessment approach in conjunction with its short courses. In part, following the systematic review of the randomized control trial literature provided by Davis, et al. (1995), we pioneered this approach believing that it would make it more likely that our courses would have greater impact. Our initial work in teaching physicians to better manage alcohol-related problems further reinforced our thinking (Lockyer, el-Guebaly, Simpson, Gromoff, & Toews, 1996). Later as the College of Family Physicians of Canada changed its accreditation criteria for programs, assigning extra value to those that included preand postcourse assessment, reflective exercises and small group (<10 professionals), we began to develop and offer outcomesbased courses in which these were standard components. Journal of Executive Education Fall 2003 33 Our outcomes-based courses take a fairly standard but consistent approach to curriculum planning and implementation. The courses are designed by physicians for physicians with at least half of the planning committee made up of physicians from the discipline for which the course is designed. This helps to ensure that the content is appropriate and contextually based. The learning objectives are congruent with the needs assessment processes. Speakers receive clear and explicit directions about content and format. Consideration is given to using learning strategies and holding programs in environments that support learning. All courses are evaluated by learners, and their assessment guides curriculum revision. To achieve the highest level of study credit, the courses are designed so that at least two-thirds of the time is spent in groups of 10 or fewer physicians. There is an individualized needs assessment prior to the course and a post-course (three months later) reflective exercise. Thus, the educational program is extended beyond the usual time frame for an educational intervention. While the processes of course development are similar, the subject matter, learning and practice needs, and the learning tasks all affect the course design. Some courses have several short modules (two hours each) that allow physicians to focus on selected aspects of care (e.g., how to diagnosis a condition or how to manage a diagnosis). Other courses are more comprehensive, covering diagnosis, treatment and follow-up as an entity. In some cases, physicians have been tracked (streamed into groups) based on clinical experience and knowledge, and placed in a group that offers a basic (beginner) approach vs. one that allows the physician to discuss and explore advanced aspects of the problem or disease. Some courses offer role-playing exercises. Other courses have included a videotape demonstration to ensure consistency in teaching or access to material that would be difficult to provide in every teaching setting. Some courses have extensive resource manuals to support learning in a complex area. Some courses have evolved substantially over time. One course is in its fourth major iteration as it changed from two long courses (eight hours) to a shorter multi modular course (three hours). Recent funding has allowed courses to be re-designed from faceto-face to asynchronous on-line formats. The approach to preand post-data collection has varied with most courses taking a similar approach. We have been guided by the course objectives, the content to be covered, the complexity of the content and its application in practice, and information about disease management we have gathered locally, nationally and sometimes internationally. Input from our teachers about how they have used the data from pre-tests to modify their teaching has also guided us. Finally, the context in which we work, a medical school that values quantitative data, randomized control trials, probability and effect size, also influences our work. Where possible we try to repeat the pre-test at the post-test phase so that we can match pairs (i.e., the same professional) of data and calculate outcomes. Generally, the post-test is conducted three months after the course. For most clinical topics, physicians will have seen a few patients with that diagnosis in the interim ensuring they have an opportunity to test the new knowledge and skills and obtain patient feedback about the new approach. We have found one month is too short. A period of six months is too long, as physician ability to recall the salient aspects of a course is limited by then. We have learned that 15 to 25 questions on a pre-test are optimal for a three-hour course. Questionnaires are not too long to discourage participation; yet, they allow physicians to think about their practices. Data provided to teachers are about right, given that many of the instruments will arrive within a few days of the course and have to be summarized. We have found that 80% to 90% of physicians will complete the pre-assessment and submit it prior to the course (Lockyer, Ward, & Toews, 1997; Ward, Fidler, Lockyer, & Toews, 1999; Ward, Fidler, Lockyer, Bassson, Elliott, & Toews, 2001; Lockyer, Fidler, Hogan, Pereles, Lebeuf, & Wright, 2002). Similarly, when we repeat the pre-assessment as part of the post-assessment, we find that 50% to 70% of physicians will 34 Fall 2003 Journal of Executive Education provide data (Lockyer et al., 1997; Ward et al., 1999; Ward et al., 2001, Lockyer et al., 2002). We use a combination of question types in the pre-test, which we term an A-B-C approach. A is for attitudes, B for behaviors, and C for cognitive knowledge. As examples of attitudes, we would query comfort level and their level of practice in a disease area. For example, one of the questions on the Alzheimer’s and Other Dementias Course instrument asks physicians how comfortable they are administering the mini mental state examination as well as questions related to comfort discussing guardianship and trusteeship. They are also asked about their involvement in the care of patients. We ask the physicians if their usual approach is 1 = “I diagnose and refer patients that I suspect have Alzheimer’s” to 5 = “I offer complete care to these patients.” Behavior is typically assessed by asking physicians to abstract data from their medical records related to the care provided to three to five patients. Physicians can provide data about the mini mental state examination scores for these patients and whether or not they actually discussed guardianship and trusteeship with the patients. This may be supplemented by having physicians tell us about the numbers of patients in their practice with Alzheimer’s or Dementia. Cognitive knowledge is assessed by multiple-choice and true/false questions. For example, with the Alzheimer’s course, one of the questions asked was, “When asked to spell WORLD backwards, the patient responded ‘LDRWO.’ Her score for this item on the MMSE is a) 0; b) 1; c) 2; d) 3; or e) 4.” Recognizing both the importance of reflection and the unique opportunity that a few minutes of silence has at the end of a course for professionals to gather their thoughts about what they have learned, we ask physicians to identify three to five items that they are going to change or adopt in their practices using a “commitment to change form” in which these statements are recorded. When the post-course assessment is sent out, physicians also receive a copy of their “commitment to change form” and are asked to provide data on the changes they actually made, difficulties they encountered making the changes and changes they did not make. The data from our examination of preand post-course information and the commitment to change statements and follow-up have guided us in the development of new courses in other areas as well as the revision of some of these courses (Lockyer et al., 1997; Ward et al., 1999; Lockyer, Fidler, Ward, Basson, Elliott, & Toews, 2001; Ward et al., 2001, Lockyer et al., 2002). We have found that these courses are feasible to design and market (Lockyer et al., 1997). Physicians will attend these courses (Lockyer et al., 1997; Ward et al., 1999; Lockyer et al., 2001; Ward et al., 2001, Lockyer et al., 2002). Physician satisfaction is high (Lockyer et al., 1997). With the appropriate marketing, some courses have had over 1,000 registrants across Canada during their 18-month life cycle. Furthermore, physicians will provide preand post-course data (Lockyer et al., 1997; Ward et al., 1999; Ward et al., 2001, Lockyer et al., 2002) as well as commitment to change statements and follow-up data (Lockyer et al., 2001). As noted earlier, physicians are more likely to provide precourse data than post-course data (Ward et al., 1999; Ward et al., 2001; Lockyer et al., 2002). Study credits are a powerful motivator. Physicians who are not certificants of the College of Family Physicians of Canada and don’t need study credits are less likely to complete the postcourse assessments (Ward et al., 1999, 2001; Lockyer et al., 2002). When the learning task has been complex and physician practices variable, physicians have been willing to be directed to the “track” or “stream” that is most appropriate based on an analysis of their individualized needs Journal of Executive Education Fall 2003 35 assessment (Lockyer et al., 1997; 2002). Furthermore, it is possible to develop methods to accurately separate physicians based on their clinical practices (Lockyer et al., 2002). Our work with paired samples (i.e., where we have data for a single physician before and after the course) shows that knowledge scores, levels of involvement in patient care, and comfort in managing salient aspects of care will improve and that this change is significant (Ward et al., 1999; Ward et al., 2001; Lockyer et al., 2002). While these data achieve statistical significance, we have found that the “effect size” will vary (Lockyer et al., 2002), possibly due to data ceiling effects for more experienced clinicians. Virtually all physicians attending courses will commit to making changes (Lockyer et al., 2001). Almost half of these physicians will provide follow-up data. Approximately two-thirds of changes originally committed to will be implemented. Commitments will be closely related to the time spent on that content in the course (Lockyer et al., 2001). There are limitations to our studies. First, we have depended on self-report feedback from physicians. We do not have mechanisms to verify these data. Second, we have noted that certain types of physicians (certificants of the College of Family Physicians of Canada vs. non certificants) are more likely to provide postcourse data. Third, we are working with naturalistic data. Physicians have not been randomly assigned to take or not take the course. For most of the educational programs, other educational interventions, collegial discussions and reading would also have taken place after the course. It is impossible to measure the outcome on practice of the course relative to other education the physicians may have undertaken. Discussion In this paper we have provided a rationale for using preand post-test assessments to extend the cognitive processes afforded by a short course. While we would like to offer more in-depth courses and courses involving supervision, the time practicing physicians have for education is limited. Our approach has been one of honoring the time commitment for a face-to-face experience while asking the physicians to focus on the course during a 2to 3-week period prior to the course. We have attempted to situate that thinking in their workplace as they review patient charts and think about their comfort with specific aspects of diagnosis, investigation, treatment or follow-up of patients. We believe that our post-course followup allows them once again to reflect on the learning activity within their current workplace situation. By providing copies of the commitment to change statements completed earlier, the physicians have a chance to assess their learning in the interim as well as set new goals and find new sources for the next phase of learning. We have been deliberate in selecting a quantitative approach rather than a qualitative approach to our data collection and analysis. In part, we work in a medical environment in which this type of data is favored. Additionally, many of our partners are looking for evidence that there has been impact on care provided. While it is possible to do this with qualitative data as our commitment to change study shows (Lockyer et al., 2001), the data analysis is more time consuming and won’t necessarily guide the revision of the program more effectively. Further, the physicians are busy and we have found that physicians are less likely to take the time to complete open-ended questions in sufficient detail to be helpful to us. For most courses, our ABC (attitude, behavior, and cognitive knowledge) approach to assessment pre and post tells us that change has occurred in the 36 Fall 2003 Journal of Executive Education desired direction. Physicians are working at a higher level, assuming more responsibility for patient care, and know more; and they tell us they do more of the things we think they should be doing after the program. Additional work is needed to continue to analyze and publish data from other courses, particularly taking a careful look at effect size, which was not done in the earlier studies (Ward et al., 1999; Lockyer et al., 2001). As these courses have varied in length, intensity, and “newness” of concepts, it will be important to analyze data related to effect size in conjunction with course type and structure. Nonetheless, this work further confirms the findings of Davis et al. (1995, 1999) that active participatory learning that takes place over time and is reinforced and enabled is more likely to result in change than didactic lecture-based educational programs. While our findings suggest applicability to medicine, testing in other professional settings and with other disciplines is needed.